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urrent  Department  of  Defense  (DoD) 
acquisition  policy  mandates  the  use  of 
integrated  testing.  The  policy  not  only 
makes  economic  sense 
but  also  has  the  potential 
to  reduce  risk,  as  early,  integrated  testing 
often  involves  more  realistic  operational 
scenarios  than  traditional  developmental 
testing  and  therefore  allows  earlier  discov¬ 
ery  of  operational  failure  modes.  As  more 
programs  have  attempted  to  implement 
the  policy,  however,  they  have  encoun¬ 
tered  obstacles  that  have  prevented  them 
from  fuUy  realizing  the  benefits  of  inte¬ 
grated  testing.  Issues  that  present  difficul¬ 
ty  in  integrated  testing  fall  into  three 
principal  areas:  sharing  and  access  to  data; 
shared  control  of  test  events;  and  overre¬ 
action  by  some  observers  to  the  test  results.  I  believe 
the  real  obstacles  to  fully  implementing  integrated 
testing  are  mostly  cultural  and  can  be  overcome  with 
appropriate  action  by  acquisition  leaders. 

DoD  policy  memos  and  guidance  documents  define 
what  we  mean  by  “integrated  testing.”  The  Defense 
Acquisition  Guidebook,  Test  and  Evaluation  (T8cE) 
chapter  (chapter  9)  provides  the  formal  definition  and 
additional  detail.  The  definition  focuses  on  collabora¬ 
tive  planning  and  execution  of  tests  to  provide  a  shared 
or  common  data  set  for  independent  evaluations  and 
reporting.  It  is  important  to  note  that  the  definition  is 
not  “integrated  test  and  evaluation”  but  “integrated 
testing!'  Although  the  testing  is  planned  and  executed 
coUaboratively  by  the  contractor,  government  Develop¬ 
mental  Test  (DT)  and  Operational  Test  (OT) 
communities,  the  evaluations  are  performed  indepen¬ 
dently  to  fulfill  respective  roles  and  missions. 

The  challenges  regarding  sharing  and  access  to  data 
seem  to  be  associated  largely  with  ensuring  the 
pedigree  of  the  data  and  proprietary  issues  with 
contractor  data.  As  defined,  integrated  testing  includes 
contractor  testing  and  can  result  in  claims  of 
proprietary  data  rights.  In  order  to  share  the  data 
from  contractor  events,  provisions  for  data  access  must 
be  included  in  program  contracts.  The  converse  is  also 
true  if  contractors  are  expected  or  allowed  to  use  data 
collected  during  government  test  events.  Discussing 


data  access  issues  up  front,  before  the  contract  is 
signed,  can  set  the  expectation  and  allow  for  an 
equitable  arrangement  for  obtaining  access  to  contrac¬ 
tor  data.  In  order  to  assist  programs  with 
these  types  of  T8cE  contractual  matters, 
my  office  has  produced  a  guide,  “Incor¬ 
porating  Test  and  Evaluation  into  De¬ 
partment  of  Defense  Acquisition  Con¬ 
tracts”  (http://www.acq.osd.mil/sse/pg/ 
guidance.html). 

Ensuring  the  pedigree  of  the  data 
refers  to  understanding  the  configura¬ 
tion  of  the  test  asset  and  the  actual  test 
conditions  under  which  each  piece  of 
data  was  obtained.  While  the  primary 
purpose  of  integrated  testing  is  to 
increase  the  value  and  efficiency  of  test 
events,  the  practice  of  sharing  data  also 
could  result  in  a  reduction  in  the  acquisition  timeline  if 
we  use  shared  data  to  satisfy  multiple  objectives.  By 
infusing  operationally  relevant  profiles  and  a  mission 
perspective  during  integrated  testing,  and  establishing 
and  maintaining  the  data  pedigree,  much  of  the  data 
needed  by  the  Operational  Test  Agency  (OTA)  could 
be  obtained  before  Initial  Operational  Test  and 
Evaluation  (lOT&E).  If  the  data  collected  during 
integrated  testing  is  adequate,  it  could  serve  to  shorten 
the  dedicated  OTScE  phase. 

The  issue  with  shared  control  of  test  events  appears 
to  be  acceptance  and  use  of  data  captured  from  test 
events  outside  the  evaluator’s  sole  control.  It  is  easy  to 
have  confidence  in  data  from  a  test  you  controlled.  If 
tests  are  planned  and  conducted  coUaboratively, 
though,  all  stakeholders  (both  DT  and  OT)  have 
control  of  the  event,  and  all  should  be  able  to  accept 
the  data  from  the  tests.  The  independence  of  the 
separate  evaluation  is  not  compromised  by  the  fact  that 
the  source  of  the  data  was  an  integrated  event. 

Another  issue  is  the  potential  overreaction  by 
observers  to  test  results.  Many  program  managers  view 
problems  discovered  during  testing  as  bad  news.  Most 
understand  that  T8cE  results  are  important  in 
maturing  the  system  design  through  the  systems 
engineering  process,  but  reports  of  problems  discovered 
in  early  testing  could  be  misinterpreted  by  outside 
observers.  The  potential  for  these  misunderstandings 
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creates  a  disincentive  to  stress  the  system  early.  It 
creates  an  incentive  to  perform  DT  in  tightly  scripted 
scenarios  in  order  to  demonstrate  successful  system 
operation.  Unfortunately,  this  conduct  merely  post¬ 
pones  the  discovery  of  operational  failure  modes, 
resulting  in  expensive  rework  and  causing  disruption 
to  program  schedules.  We  need  the  acquisition  and 
user  communities  to  realize  that  the  product  of  TScE  is 
knowledge  about  the  system’s  capabilities  and/or 
limitations,  not  problems.  Testers  and  evaluators  must 
develop  knowledge  that  is  relevant  and  timely  for  the 
decisions  being  made,  and  report  results  in  a  mission- 
oriented  context.  Ultimately,  we  need  comprehensive 
knowledge  from  T&E  results  to  assist  in  managing 
risks  and  better  decision  making. 

Integrated  testing  holds  a  promise  of  greater  testing 
efficiencies  and  improving  the  quality  of  the  informa¬ 
tion  provided  to  the  decision  makers.  The  challenge  to 
the  TScE  community  is  to  implement  robust  integrated 
testing  and  change  the  culture  to  fully  realize  the 
benefits  to  the  acquisition  process  and  ultimately  to  the 
warfighters.  □ 
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